Structural Properties of Stochastic Dynamic Programs

In Markov models of sequential decision processes, one is often interested in showing that the value function is monotonic, convex, and/or supermodular in the state variables. These kinds of results can be used to develop a qualitative understanding of the model and characterize how the results will change with changes in model parameters. In this paper we present several fundamental results for establishing these kinds of properties. The results are, in essence, “metatheorems” showing that the value functions satisfy property P if the reward functions satisfy property P and the transition probabilities satisfy a stochastic version of this property. We focus our attention on closed convex cone properties, a large class of properties that includes monotonicity, convexity, and supermodularity, as well as combinations of these and many other properties of interest.